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Background / Context: 

Experimental approaehes ean help disentangle the impaets of polieies from the effeets of 
individual eharaeteristies, but the heterogeneity of implementation inherent in studies with 
eomplex program designs may mask average treatment impaets (Morris & Hendra, 2009). In the 
ease of the Opportunity NYC-Family Rewards (ONYC-Family Rewards), families earned a 
range of differing incentives, as a result of variation in engagement in the activities rewarded by 
the program. Given that the effect of ONYC-Family Rewards on key domains of interest 
(education, health, and employment) are thought to occur through such engagement (as well as 
the increases in income that accompany it), this heterogeneity is critical to explore. 

Purpose / Objective / Research Question / Focus of Study: 

In this study, we examine the extent to which effects of ONYC-Family Rewards are 
moderated by key family and child- level characteristics that are associated with a families’ 
propensity to earn the rewards offered by the program. We use multiple baseline characteristics 
to predict the amount of dollars earned from the rewards as a means to identify theoretically 
important, multivariate-defined groups of children for whom program effects might be more 
concentrated. Specifically, we ask: (1) Does the likelihood of earning rewards moderate the 
impact of ONYC-Family Rewards on children’s educational outcomes? (2) Are the impacts of 
ONYC-Family Rewards on children’s educational outcomes greater among the group of children 
whose families are most likely to earn rewards? 

Setting: 

The intervention was aimed at low- income families in six of New York City’s highest- 
poverty communities in the Bronx, Brooklyn and Manhattan. 

Population / Participants / Subjects: 

The sample includes 4,750 families and over 1 1,000 children, half of whom were eligible 
to receive the cash incentives if they met the required conditions. Children (50 % female; 
average age at time 1 = 11 yrs) were spread across 900 schools in New York City. The majority 
of children were attending a public or charter school (97.5 %) in the year prior to the study, while 
a small number were attending a private or parochial school (2.5 %). Most children were 
Hispanic/Fatino (47.0 %) and Black, non-Hispanic/Fatino (50.4 %) and were almost all born in 
the U.S. (93.5%). A number of children (14.9 %) were enrolled in special education, while 12.9 
% of children were enrolled as an English Fanguage Feamer (EFF) in the year prior to the study. 

Families recruited into the ONYC-Family Rewards study had to have at least one child in 
4*, 7^'', or 9* grade at the start of the study, based on Department of Education (DOE) 
administrative records in the year pre-random assignment. This child was treated as the focal 
child in the family. If families had more than one child in the target grade, one child was 
randomly selected to be the focal child. From this sample of families with a target child, we 
created three separate analysis samples used in our impact analyses. The youngest age group was 
made up of families with a focal child in elementary school, expected to be in 4* grade at the 
start of the study (N = 1451). The middle age group was made up of families with a focal child in 
middle school, expected to be in 7**^ grade at the start of the study (N = 1370). The oldest age 
group was made up of families with a focal child in high school, expected to be in 9* grade at the 
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start of the study (N = 1550). We will be reporting effects of ONYC on a subgroup of these 
families who earned the most amount of money from rewards. This sample includes 369 4**^ 
grade children, 339 7^*' grade children, and 389 9^'' grade children. 

In addition, we created an external modeling sample for each age group from the 
subsample of all families assigned to the ONYC program who had a child in 3’^'*, 4*, or 5* grade 
who was not a focal child. For the youngest age group, our external modeling sample (N = 528) 
included a subsample of all families who had a child in 3'^‘*, 4^*', or 5* grade who was not a focal 
child. For the middle age group, the modeling sample (N = 487) included a subsample of all 
families who had a child in 6*, 7*, or 8* grade but who was not a focal child. For the 9**^ grade 
age group, the modeling sample (N = 370) included a subsample of families who had a child in 
8* or 9**^ grade but who was not a focal child. Creating a modeling sample for each age group 
that parallels the three analysis samples ensures that the children and families in the modeling 
sample in each age group resemble the children and families in the analysis sample. 

Intervention / Program / Practice: 

Opportunity NYC-Family Rewards is a conditional cash transfer (CCT) program that ties 
cash rewards to a pre-specified set of activities and outcomes in the areas of children’s education, 
family preventive health care, and parents’ employment. Reflecting the important role that parents 
can play in their children’s success in school, the incentives in this domain are intended to encourage 
parents to become more fully engaged with their children’s education, (see detailed description in 
Abstract 1 in this symposium). 

Research Design: 

This analysis, building from a technique used in propensity score research but used here 
to assess subgroup impacts in experimental studies, allows us to maintain the integrity of the 
experiment while identifying the group of families who were most likely to earn rewards (Morris 
& Hendra, 2009). This technique differs from a propensity score matching approach in that we 
do not identify a control group for comparison purposes (Deheja & Wahba, 1999; Rosenbaum & 
Rubin, 1983, 1984). Instead, this approach is used to develop an index that can be used along 
with the randomization design to estimate how the effects of the program differ across families at 
differing levels of the rewards index. 

Data Collection and Analysis: 

The data collected spans a vast range of sources and includes administrative records on 
school outcomes, employment, earnings, public health insurance, welfare and food stamp payments, 
and housing subsidies obtained from various New York City and New York State agencies; program- 
related data on reward payments obtained from Seedco; and one wave of a survey in which a subset 
of parents in the program and control groups are interviewed 1 8 months after treatment began. The 
parent-survey, administered by the Department of Information Resources (DIR), was administered 
via telephone using CATl and had an overall response rate of 82 %. As part of the survey, 
parents were asked to report on the 4*, 7*, or 9* grade child who was previously identified as the 
focal child. Data on education outcomes were obtained from administrative records provided by 
the New York City Department of Education (DOE). These data are available for all students in 
the study for one year prior to study entry and for two years after study entry. School outcomes 
that are available from the DOE records include attendance rates, scores on annual math and 
English language arts (ELA) tests, performance on Regents exams, course credits earned, and 
school enrollment status. 
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In these analyses we use an external modeling sample that is subsequently removed from 
the analysis when estimating program impaets. Using a weighted, regression-based approaeh, we 
ereated an index in the modeling sample based on multiple, theoretieally-derived, family and 
ehild baseline eharaeteristies to predict the amount families earned from the rewards. The 
baseline characteristics included marital status, number of children in the family under 19, age, 
foreign born status, race, presence of a child between the ages of 0 and 2, presence of a child 
between the ages of 13 and 18, TANF receipt, health insurance receipt, whether or not the parent 
has a mental or physical problem limiting work, full-time employment status of parent and of 
any adult in the household, benefit or housing subsidy receipt, parents’ education level, parents’ 
earnings in the year prior to random assignment, child gender, child special education status, 
child math and reading achievement in the year prior to random assignment, and child attendance 
rate in the year prior to random assignment. 

Using the index, we identified the top quartile of families on the index, in both the 
program and the control groups. We then estimated experimental impacts in the first two years 
on children and their families among this subgroup using OLS regression, as well as the 
moderating effects of the index on the effects of the treatment indicator. Differences in subgroup 
impacts were tested by conducting split sample regression analyses and estimating differences 
using an HT statistic. The HT statistic is the weighted sum of squares of the impact estimates for 
the subgroups and has a chi-squared distribution (Cooper & Hedges, 1994). Unlike more 
standard interaction terms, split sample approaches have the advantage of not assuming 
homogeneity of variance across the subgroups examined. 

Findings / Results: 

Recall that the core study findings showed no statistically significant benefits of the 
program on educational outcomes for the youngest children (see Abstract 1). Preliminary 
analyses conducted here find statistically significant program benefits on a number of measures 
of academic achievement among the youngest children whose families were likely to earn the 
most money from the rewards. As shown in Table 1, for this group of children, the program led 
to a statistically significant increase in attendance and in Math and Reading standardized test (in 
Year 1). There were no effects among the 7* and 9**^ grade cohorts for this same subgroup of 
families. 

Conclusions: 

Although these findings are preliminary, they suggest that the program may be improving 
academic outcomes for those youngest children whose families are likely to experience the 
greatest increase in reward income. Further analyses will explore the mediating processes 
underlying these effects, to gain further confidence in these results. 
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Table 1. Impacts on Attendance and Achievement among Families with the Highest Likelihood of Earning Rewards by 
Child Age Group 



4th Grade Children 


7th Grade Children 


9th Grade Children 




Control 


Difference 


Control 


Difference 


Control 


Difference 


Outcome (%) 


Group 


(Impact) 


Group 


(Impact) 


Group 


(Impact) 


Attendance 










■1 




Had attendance rate of 95% or higher, Year 1 


0.62 


0.02 


0.71 


-0.05 




0.03 


Had attendance rate of 95% or higher, Year 2 


0.53 


0.10 * 


0.57 


-0.03 




0.07 


Attendance rate. Year 1 


0.95 


0.00 * 


0.96 


0.00 


0.92 


-0.02 


Attendance rate. Year 2 


0.89 


0.04 * 


0.92 


-0.02 


0.89 


0.01 


Achievement 














Math Scaled score. Year 1 


683.00 


6.00 *' 


668.61 


0.01 






Math Scaled score. Year 2 


684.73 


5.52 


674.62 


-0.66 






Reading Scaled score. Year 1 


658.68 


5.21 


658.68 


-0.49 






Reading Scaled score. Year 2 

Took at least 2 Regents exams. Years 1 


668.34 


2.92 


656.07 


0.18 


0.52 


0.00 


Passed at least 2 Regents exams. Years 1 










0.39 


-0.05 


Took at least 2 Regents exams. Years 2 










0.80 


0.04 


Passed at least 2 Regents exams. Years 2 










0.50 


0.04 


Sample size 


183 (total = 367) 


175 (total = 335) 


180 (total = 389) 



SOURCE: MDRC calculations using data from New York City Department of Education administrative records. 



NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. Differences across 
subgroup impacts were tested for statistical significance. The high risk subgroup was determined by taking the group of families 
who fell in the top 25th percentile of the take up index distribution. 

Sample sizes vary because of missing values. 
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